Imaging method and imaging device

ABSTRACT

Disclosed herein is an imaging method including: by an imaging device, detecting a user that can be present in a predetermined area; calculating three-dimensional coordinates of an eye and a hand of the detected the user; determining an imaging range on a basis of the calculated the three-dimensional coordinates of the eye and the hand of the user; and imaging the determined the imaging range.

BACKGROUND

The present disclosure relates to an imaging method and an imagingdevice, and particularly to an imaging method and an imaging device thatenable photography using a camera installed in a photographing spot of atourist resort, for example.

When a person visits a place as a so-called photographing spot in atourist resort or the like, the person needs to carry a camera or thelike with himself or herself to take a photograph on the spot.

In addition, even when a person carries a camera, in order to take agroup photograph of a group to which the person belongs on the spot, itis for example necessary to ask a person outside the group to perform ashutter operation, or necessary for someone within the group to performa shutter operation without being photographed in the group photograph.In addition, a timer function or a remote shutter function of a cameramay be used.

A method for enabling a commemorative photograph to be taken using acamera installed in advance in a tourist resort or the like has beenproposed for a case of not carrying a camera (see Japanese PatentLaid-Open No. 2004-282769, for example).

SUMMARY

However, an invention described in Japanese Patent Laid-Open No.2004-282769 enables photographing only with a predetermined composition(a direction, a range and the like of the photographing), and a usercannot determine the composition of a photograph.

The present disclosure has been made in view of such a situation. It isdesirable to enable a photograph taken with an arbitrary composition tobe obtained without a camera being carried.

According to an embodiment of the present disclosure, there is providedan imaging method including: by an imaging device, detecting a user thatcan be present in a predetermined area; calculating three-dimensionalcoordinates of an eye and a hand of the detected user; determining animaging range on a basis of the calculated three-dimensional coordinatesof the eye and the hand of the user; and imaging the determined imagingrange.

The imaging range can be determined by projecting three-dimensionalcoordinates of a rectangle identified by fingers of both hands of theuser from the three-dimensional coordinates of the eye of the user.

An imaging direction can be determined on a basis of the calculatedthree-dimensional coordinates of the eye and the hand of the user, andan optimum composition set in advance for the determined imagingdirection can be set as the imaging range.

The imaging method according to the embodiment of the present disclosurecan further include: by the imaging device, searching images stored inadvance for a similar image similar to a picked-up image obtained as aresult of the imaging; and correcting the picked-up image using thesimilar image.

The imaging method can further include, by the imaging device,identifying a group member belonging to a same group as the user amongsubject persons in the picked-up image obtained as the result of theimaging, wherein a region of a subject person other than the groupmember in the picked-up image can be replaced using the similar image.

According to another embodiment of the present disclosure, there isprovided an imaging device including: a detecting section detecting auser that can be present in a predetermined area; a calculating sectioncalculating three-dimensional coordinates of an eye and a hand of thedetected user; a determining section determining an imaging range on abasis of the calculated three-dimensional coordinates of the eye and thehand of the user; and an imaging section imaging the determined imagingrange.

In the embodiment of the present disclosure, a user that can be presentin a predetermined area is detected, three-dimensional coordinates of aneye and a hand of the detected user are calculated, an imaging range isdetermined on a basis of the calculated three-dimensional coordinates ofthe eye and the hand of the user, and the determined imaging range isimaged.

According to the embodiment of the present disclosure, it is possible toobtain a photograph taken with an arbitrary composition without carryinga camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of configuration of anautomatic imaging system to which embodiments of the present disclosureis applied;

FIG. 2 is a diagram showing relation between the eye and both hands ofthe user and the composition of a photograph;

FIGS. 3A, 3B, and 3C are diagrams showing an example of patterns of theposture of hands by a user;

FIG. 4 is a flowchart of assistance in explaining imaging processing;

FIG. 5 is a flowchart of assistance in explaining imagingpost-processing;

FIGS. 6A, 6B, and 6C are diagrams showing an example of erasing subjectpersons other than group members; and

FIGS. 7A, 7B, 7C, and 7D are diagrams showing another example of erasingsubject persons other than group members.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The best mode (hereinafter referred to as an embodiment) for carryingout the disclosure will hereinafter be described in detail withreference to the drawings.

1. Embodiments [Example of Configuration of Automatic Imaging System]

FIG. 1 shows an example of configuration of an automatic imaging systemas an embodiment of the present disclosure. In this automatic imagingsystem, in response to a predetermined posture (to be described later)indicating the composition of a photograph which posture is assumed by aperson not carrying a camera (which person will hereinafter be referredto as a user) in a so-called photographing spot such as a tourist resortor the like, a camera installed on the periphery takes a photograph withthe composition (hereinafter referred to also as an imaging range)intended by the user.

The automatic imaging system 10 includes a plurality of cameras 11-1 to11-3 connected to each other via a network 12, an imaging managingdevice 13, and an image managing device 14.

The cameras 11-1 to 11-3 are installed in a place that can be aphotographing spot. Specifically, the cameras 11-1 to 11-3 are mountedon rails having a predetermined length, for example, and are moved onthe rails, whereby the installation positions of the cameras 11-1 to11-3 can be changed. The cameras 11-1 to 11-3 can also be changed inimaging direction (an azimuth angle and an angle of elevation) and zoommagnification.

The camera 11-1 picks up an image (hereinafter referred to as a firstimage) for detecting a user and measuring the three-dimensionalcoordinates of an eye and both hands of the user. The camera 11-1performs consecutive photographing in a direction in which a user can bepresent in the photographing spot. The camera 11-1 outputs the firstimage obtained as a result of the consecutive photographing to theimaging managing device 13 at any time.

The camera 11-2 picks up a second image for measuring a distance betweenthe user and a subject person standing in front of a background such asscenery, a building or the like of the photographing spot. The camera11-2 performs photographing in a direction in which the user and thesubject person in the photographing spot can both be photographed. Thecamera 11-2 outputs the second image obtained as a result of thephotographing to the imaging managing device 13.

The camera 11-3 images a photographing range specified by the imagingmanaging device 13 in predetermined sampling cycles. The camera 11-3outputs an image (hereinafter referred to as a third image) obtained asa result of the imaging to the imaging managing device 13. In addition,the camera 11-3 images a photographing range specified by the imagingmanaging device 13 in specified timing. The camera 11-3 outputs an image(hereinafter referred to as a picked-up image) obtained as a result ofthe imaging to the imaging managing device 13 together with imagingconditions (the coordinates of the installation position, the imagingdirection, and the zoom magnification of the camera 11-3).

Further, the camera 11-3 periodically picks up images as candidates fora similar image (to be described later in detail) to be combined withthe picked-up image under varying conditions such as composition,weather, season, time, and the like. The camera 11-3 supplies theobtained images and imaging conditions at the time of the image pickupto the image managing device 14 via the network 12.

Suppose that the imaging conditions of the cameras 11-1 to 11-3 (thecoordinates of the installation position, the imaging direction, and thezoom magnification of the camera 11) are notified to the imagingmanaging device 13 via the network 12, and are grasped by the imagingmanaging device 13 at all times.

In addition, while the purposes of the cameras 11-1 to 11-3 aredifferentiated from each other for convenience in the above description,each of the cameras 11-1 to 11-3 can supply the purpose of the other orconcurrently serve the purpose of the other.

The cameras 11-1 to 11-3 do not need to be provided exclusively for theautomatic imaging system 10. Surveillance cameras, observation cameras,and the like that are already installed may be used as the cameras 11-1to 11-3.

The imaging managing device 13 includes an image recognizer that hasalready performed machine learning of images of faces, eyes, hands, andthe like of humans. By using the image recognizer, the imaging managingdevice 13 detects a person as the user from the first image input fromthe camera 11-1 via the network 12 at any time. Specifically, forexample, the imaging managing device 13 regards and detects a personassuming a predetermined posture (with fingers of both hands shown inFIG. 2 forming respective L-shapes and thereby indicating a rectangle)as the user.

Alternatively, for example, a mark (for example a badge or a sticker)indicating a user may be given to a user in advance to be attached tothe chest or the like of the user, and the person to which the mark isattached may be detected as the user.

The imaging managing device 13 retains three-dimensional mapping data onranges that can be photographed by the respective cameras 11-1 and 11-2.

FIG. 2 shows relation between the eye and both hands of the user and theimaging range (referred to also as composition, as described above)imaged by the camera 11-3.

Suppose that the user assumes a posture such that fingers of both handsform respective L-shapes and thereby indicate opposite vertices of arectangle as an imaging range in an imaging direction in order tospecify the imaging range, and that the user performs a predeterminedshutter operation (for example bends the tip of an index finger, forexample) to specify imaging timing.

When the user assumes a posture as described above, thethree-dimensional coordinates of the eye 21 of the user are calculated.Next, the three-dimensional coordinates of a total of four points, thatis, the three points of the L-shape formed by fingers of one of the lefthand 22L and the right hand 22R of the user and the vertex of theL-shape formed by fingers of the other hand of the user are calculated,and the three-dimensional coordinates of the four vertices of therectangle formed by the hands of the user are identified. Further, animaging range 23 is determined by performing perspective projection ofthe four vertices of the rectangle formed by the hands of the user withthe position of the eye 21 of the user as a reference.

Incidentally, the imaging managing device 13 may retain, in advance,several kinds of ideal composition for photographs to be taken in thephotographing spot in which the cameras 11-1 to 11-3 are installed, andone of the several kinds of composition that coincides with thedirection indicated by the hands of the user may be used as the imagingrange 23.

It suffices to determine the ideal composition through the compositionanalysis of a plurality of images picked up on the spot, or to allow theplurality of images to be downloaded and determine the ideal compositionon the basis of the numbers of downloads of the plurality of images, orto allow votes to be cast about the evaluation of the plurality ofimages and determine the ideal composition on the basis of a result ofthe voting. Alternatively, the ideal composition may be determined fromthese results taken together.

The use of the ideal composition as the imaging range 23 provides anadvantage for the user in that an image of good composition can beobtained by merely roughly specifying a photographing direction. Thereis also an advantage on the side of the automatic imaging system 10 inthat an imaging range can be determined even when the positions of theeye and the hands of the user cannot be recognized accurately due to theposition or the imaging direction of the camera 11 or the like.

Returning to FIG. 1, the imaging managing device 13 identifies thepositions of the eye and both hands of the user from the first image byusing the image recognizer, and calculates the three-dimensionalcoordinates of the eye and both hands of the user on the basis of thethree-dimensional mapping data corresponding to the first image.Further, the imaging managing device 13 calculates a distance betweenthe user and a subject person on the basis of the second image inputfrom the camera 11-2 and the three-dimensional mapping datacorresponding to the second image. Further, the imaging managing device13 determines an imaging range 23 (FIG. 2) on the basis of thecalculated three-dimensional coordinates of the eye and both hands andthe calculated distance between the user and the subject person, andnotifies the imaging range 23 to the camera 11-3.

Incidentally, instead of the imaging managing device 13 retaining thethree-dimensional mapping data as described above, the cameras 11-1 and11-2 may each be replaced with a stereo camera, so that the distance tothe subject can be measured on the basis of stereo matching.Alternatively, a range sensor may be incorporated into the cameras 11-1and 11-2. It is thereby possible to calculate the three-dimensionalcoordinates of the eye and both hands of the user and calculate thedistance between the user and the subject person without using thethree-dimensional mapping data.

The imaging managing device 13 estimates persons belonging to a samegroup as the user (which persons will hereinafter be referred to asgroup members) among the subject persons appearing in the third imageinput from the camera 11-3 on a sequential basis, and generatespre-imaging group information indicating a result of the estimation.Specifically, the imaging managing device 13 extracts the eyes of facesof persons from the third image by face recognition processing, andestimates that persons whose directions of lines of sight continue to bethe direction of the user for a certain time are group members. Thegenerated pre-imaging group information is associated with informationfor identifying the user (which information will hereinafter be referredto as user information), and then notified to the image managing device14 via the network 12 such as the Internet and the like.

In addition, when the imaging managing device 13 detects a predeterminedshutter operation (for example bending the tip of a finger forming anL-shape) by the user on the basis of the first image, the imagingmanaging device 13 makes the camera 11-3 image the imaging range 23 insynchronism with the timing of the shutter operation, and therebyobtains a picked-up image. Incidentally, in a timer mode to be describedlater, it suffices to effect the imaging in timing delayed by apredetermined time from the timing of the shutter operation. Theobtained picked-up image is associated with the user information, andthen output to the image managing device 14 via the network 12.Incidentally, a third image coinciding with the timing of the shutteroperation among third images input in predetermined sampling cycles maybe regarded as the picked-up image.

The image managing device 14 analyzes the picked-up image input from theimaging managing device 13 via the network 12, estimates group membersamong the subject persons appearing in the picked-up image, andgenerates post-imaging group information indicating a result of theestimation.

Specifically, as in the generation of the pre-imaging group information,the eyes of faces of persons are extracted from the picked-up image byface recognition processing, and it is estimated that persons whosedirections of lines of sight are the direction of the user are groupmembers. In addition, the accuracy of the post-imaging group informationmay be increased by also estimating group members in another picked-upimage picked up in series with the picked-up image in question whichgroup members correspond to the identical user and regarding onlypersons repeatedly estimated to be group members in the plurality ofpicked-up images as group members.

Further, the image managing device 14 finally judges only subjectpersons repeatedly estimated to be group members among the subjectpersons according to both of the pre-imaging group information and thepost-imaging group information to be group members, associates the facesof the group members with the user information, and registers the facesof the group members associated with the user information in a group DB(database) managed by the image managing device 14 itself.

Further, the image managing device 14 searches for an image (hereinafterreferred to as a similar image) similar to the picked-up image inquestion from an image DB managed by the image managing device 14 bycomparing the coordinates of the installation position, the imagingdirection, and the zoom magnification of the camera 11-3 and thecomposition.

Incidentally, suppose that the image DB stores a plurality of imagespicked up by the camera 11-3 under different conditions of composition,weather, season, time, and the like in association with the coordinatesof the installation position, the imaging direction, and the zoommagnification of the camera 11-3. Incidentally, because a region inwhich no person is photographed is used as a similar image, as will bedescribed later, it is desirable that no person be photographed in thesimilar image. It is difficult, however, to pick up a similar image intiming in which no person is photographed in the photographing spot.Accordingly, regions in which no person is photographed among a largenumber of images picked up under similar conditions (composition,weather, season, time, and the like) may be combined with each other togenerate a similar image in which no person is photographed, and storethe similar image.

The image managing device 14 generates a picked-up image in a state ofpersons other than the group members being erased by replacing a regionin which persons other than the group members are photographed in thepicked-up image with a corresponding region in which no person isphotographed in a similar image. The picked-up image in the state of thepersons other than the group members being erased is stored in the imageDB in association with the user information. The picked-up image in thestate of the persons other than the group members being erased can bedisplayed or downloaded when the user accesses the image managing device14 via the network 12 using a portable telephone, a computer, or thelike, for example.

[Changing Imaging Mode]

FIGS. 3A, 3B, and 3C show various patterns of the posture of both handsof the user at a time of specifying an imaging range 23. An imaging modemay be changed according to these various patterns.

For example, as shown in FIG. 3A, when the user forms an L-shape with athumb and another finger (an index finger or the like) with the backs ofboth hands facing the user himself or herself, a normal mode (in whichimaging is performed once immediately in response to a shutteroperation) is set.

In addition, for example, as shown in FIG. 3B, when the user forms anL-shape with a thumb and two other fingers (an index finger and a middlefinger or the like) with the backs of both hands facing the user himselfor herself, a rapid shooting mode (in which imaging is performed aplurality of times in response to a shutter operation) is set.

Further, for example, as shown in FIG. 3C, when the user faces the palmsof both hands toward the user himself or herself, a timer mode (in whichimaging is performed a few seconds after a shutter operation) is set.When the timer mode is used, the user can move to an imaging range 23 tobe a subject person.

The patterns of the posture of the hands shown in FIGS. 3A to 3C and theimaging modes corresponding to the respective patterns are an example,and various combinations are possible.

[Description of Operation]

Description will next be made of the operation of the automatic imagingsystem 10. The operation of the automatic imaging system 10 includesimaging processing by the cameras 11-1 to 11-3 and the imaging managingdevice 13 and imaging post-processing by the image managing device 14.

FIG. 4 is a flowchart of assistance in explaining the imagingprocessing.

This imaging processing continues being performed repeatedly while theautomatic imaging system 10 is operated.

In step S1, the imaging managing device 13 monitors a first image inputat any time from the camera 11-1 via the network 12. In step S2, theimaging managing device 13 determines whether a user is detected fromthe first image. When it is not determined that a user is detected, theprocessing is returned to step S1 to continue monitoring the firstimage. When it is determined in step S2 that a user is detected, theprocessing is advanced to step S3.

In step S3, the imaging managing device 13 detects the positions of theeye and both hands of the user from the first image. In step S4, theimaging managing device 13 calculates the three-dimensional coordinatesof the eye and both hands of the detected user. Further, the imagingmanaging device 13 in step S5 calculates a distance between the user anda subject person on the basis of the second image input from the camera11-2 and the three-dimensional mapping data corresponding to the secondimage, determines an imaging range 23 on the basis of thethree-dimensional coordinates of the eye and both hands of the user andthe distance between the user and the subject person, and notifies theimaging range 23 to the camera 11-3.

According to this notification, in step S6, the camera 11-3 images thenotified imaging range in predetermined sampling cycles, andsequentially outputs third images obtained as a result of the imaging tothe imaging managing device 13. The imaging managing device 13 estimatesgroup members from among subject persons photographed in the thirdimages sequentially input from the camera 11-3, and generatespre-imaging group information indicating a result of the estimation. Inaddition, the imaging managing device 13 determines the pattern of theposture of the hands of the user from the first image, and determines animaging mode.

In step S7, the imaging managing device 13 monitors the first imageinput from the camera 11-1, and stands by until the imaging managingdevice 13 detects a predetermined shutter operation by the user. When apredetermined shutter operation by the user is detected, the processingis advanced to step S8.

In step S8, the imaging managing device 13 makes the camera 11-3 imagethe imaging range 23 according to the determined imaging mode, andthereby obtains a picked-up image. The imaging managing device 13associates the obtained picked-up image and the generated pre-imaginggroup information with user information, and transmits the picked-upimage and the pre-imaging group information associated with the userinformation to the image managing device 14 via the network 12. Theimaging processing is then ended.

FIG. 5 is a flowchart of assistance in explaining the imagingpost-processing. This imaging post-processing is started when the imagemanaging device 14 receives the picked-up image and the pre-imaginggroup information associated with the user information, the picked-upimage and the pre-imaging group information associated with the userinformation being transmitted from the imaging managing device 13.

In step S11, the image managing device 14 analyzes the picked-up imagefrom the imaging managing device 13, estimates group members amongsubject persons photographed in the picked-up image, and generatespost-imaging group information indicating a result of the estimation.

In step S12, the image managing device 14 finally determines that onlysubject persons repeatedly estimated to be group members among thesubject persons in the picked-up image according to both of thepre-imaging group information and the post-imaging group information aregroup members.

In step S13, the image managing device 14 searches the image DB managedby the image managing device 14 itself for a similar image similar tothe picked-up image in question. In step S14, the image managing device14 for example replaces a region in which the persons other than thegroup members are photographed in the picked-up image as shown in FIG.6A with a corresponding region in which no person is photographed in thesimilar image as shown in FIG. 6B, and thereby generates a picked-upimage in a state of the persons other than the group members beingerased as shown in FIG. 6C. The imaging post-processing is then ended.

[Another Example of Erasing Subject Persons Other than Group Members inPicked-Up Image]

FIGS. 7A, 7B, 7C, and 7D show another example of erasing subject personsother than group members in a picked-up image using a similar image.

As shown in the figures, a plurality of images in which only members ofgroups (group members X and group members Y) are respectively presentmay be generated from one picked-up image.

As described above, according to the automatic imaging system 10, a userin a tourist resort or the like can obtain a photograph imaged in adesired direction without carrying a camera with the user.

In addition, a photograph with a composition which photograph could notnormally be taken by an individual can be obtained by installing acamera at a high place or installing a camera in a usually off-limitsarea. In addition, a photograph of high image quality which photographcould not normally be taken by an individual can be obtained when ahigh-performance camera as used for professional use, for example, isinstalled.

Further, the imaging managing device 13 may retain, in advance, severalkinds of ideal composition for photographs to be imaged in thephotographing spot in which the cameras 11-1 to 11-3 are installed, andone of the several kinds of composition in an imaging directionindicated by the hands of the user may be used as the imaging range 23.

It suffices to determine the ideal composition through the compositionanalysis of a plurality of images picked up on the spot, or to allow theplurality of images to be downloaded and determine the ideal compositionon the basis of the numbers of downloads of the plurality of images, orto allow votes to be cast about the evaluation of the plurality ofimages and determine the ideal composition on the basis of a result ofthe voting. Alternatively, the ideal composition may be determined fromthese results taken together. Thereby, even a user not good at takingphotographs can image a photograph of good composition (obtain an imageof good composition). In addition, the imaging range 23 can bedetermined even when the three-dimensional coordinates of the eye andthe hands of the user cannot be calculated accurately.

In the present specification, a system refers to an apparatus as a wholeformed by a plurality of devices.

It is to be noted that embodiments of the present disclosure are notlimited to the foregoing embodiments, and that various changes can bemade without departing from the spirit of the present disclosure.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2010-274231 filed in theJapan Patent Office on Dec. 9, 2010, the entire content of which ishereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors in so far as they arewithin the scope of the appended claims or the equivalents thereof.

1. An imaging method comprising: by an imaging device, detecting a userthat can be present in a predetermined area; calculatingthree-dimensional coordinates of an eye and a hand of the detected saiduser; determining an imaging range on a basis of the calculated saidthree-dimensional coordinates of the eye and the hand of said user; andimaging the determined said imaging range.
 2. The imaging methodaccording to claim 1, wherein said imaging range is determined byprojecting three-dimensional coordinates of a rectangle identified byfingers of both hands of said user from the three-dimensionalcoordinates of the eye of said user.
 3. The imaging method according toclaim 1, wherein an imaging direction is determined on a basis of thecalculated said three-dimensional coordinates of the eye and the hand ofsaid user, and an optimum composition set in advance for the determinedimaging direction is set as said imaging range.
 4. The imaging methodaccording to claim 2, further comprising: by said imaging device,searching images stored in advance for a similar image similar to apicked-up image obtained as a result of said imaging; and correctingsaid picked-up image using said similar image.
 5. The imaging methodaccording to claim 4, further comprising: by said imaging device,identifying a group member belonging to a same group as said user amongsubject persons in the picked-up image obtained as the result of saidimaging; wherein a region of a subject person other than the groupmember in said picked-up image is replaced using said similar image. 6.The imaging method according to claim 3, further comprising: by saidimaging device, searching images stored in advance for a similar imagesimilar to a picked-up image obtained as a result of said imaging; andcorrecting said picked-up image using said similar image.
 7. The imagingmethod according to claim 6, further comprising: by said imaging device,identifying a group member belonging to a same group as said user amongsubject persons in the picked-up image obtained as the result of saidimaging; wherein a region of a subject person other than the groupmember in said picked-up image is replaced using said similar image. 8.An imaging device comprising: a detecting section detecting a user thatcan be present in a predetermined area; a calculating sectioncalculating three-dimensional coordinates of an eye and a hand of thedetected said user; a determining section determining an imaging rangeon a basis of the calculated said three-dimensional coordinates of theeye and the hand of said user; and an imaging section imaging thedetermined said imaging range.